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Abstract 

The present study presents an example using a university 
student satisfaction survey data to demonstrate how to 
address problems associated with the hierarchical or nested 
nature of the data. Massive large-scale secondary data 
have been used in higher education research. Ignoring 
hierarchically structured data may lead to inaccurate or 
misleading conclusions. 
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A Hierarchical Linear Modeling Approach to Higher Education 
Research: The Influences of Student and Institutional 

Characteristics 



The majority of educational data has the hierarchical 
multilevel character. It has been the methodological 
dilemmas to use the hierarchical nature of data in 
educational research (Burnstein, 1980; Burnstein et al. , 
1989; Murchan and Sloane, 1994). Burnstein (1980) argued 
that single level traditional linear OLS approaches are not 
adequate for estimating the effects of schooling on 
students. Since Burnstein' s (1980) seminal paper, HLM has 
been used especially on the research on school effects at 
the elementary and secondary level . The researchers had 
acknowledged the hierarchical nature of the organization of 
schooling by gathering data on students, classes, and 
schools (Burnstein, 1980; Burnstein et al . , 1989; Murchan 
and Sloane, 1994) . Research in higher education had long 
been neglecting the hierarchical or nested nature of the 
data. 

In fact, many higher education models for student 
learning or cognitive development include student 
background, structural, organizational, and environmental 
characteristics of the institution, the major department, 
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and peer groups (Pascarella, 1985; Weidman, 1989). The 
models reflect the hierarchical nature of the influences on 
college students' growth and development and research has 
involved data with nested structures (Ethington, 1997) . 
Research utilizes either multi-institutional samples of 
students or single-institutional samples of students. In 
multi-institutional studies, both student and institutional 
measures are considered as factors impacting student 
outcomes. On the other hand, in single-institution 
studies, students are nested in classes, in majors, and in 
departments, schools or colleges within the institution. 

Today, large-scale secondary data sets are available 
for research in higher education. The National Center for 
Education Statistics (NCES) in the U.S. Department of 
Education produces numerous surveys such as High School and 
beyond (HS & B) , the National Postsecondary Student Aid 
Study (NPSAS) , the National Study of Post secondary Faculty 
(NSOPF) , the beginning Postsecondary student Longitudinal 
Study (BPS) , the Baccalaureate and Beyong Study (B & B) , 
and the Integrated Postsecondary educational Data System 
(IPEDS) . ’ These large data are constructed using complex 
sample designs where the population is stratified on a 
number of dimensions (Thomas & Heck, 2001) . These complex 
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sample designs often cluster lower level units (e.g., 
students) within higher-level units (e.g., colleges). 

One of the key challenges for higher education 
institutions is to arrange their resources for learning so 
that students spend their time on the activities for their 
education. Higher education institutions can continually 
examine students' perceptions of campus life to identify 
their level of satisfaction with the campus (Astin, 1975; 
Pace, 1984; Tinto, 1993) . Over the past few decades, a 
number of different tools have been developed and used to 
explore student perceptions of the quality of campus life 
(Austin, 1975; Pace, 1984). Upcraft and Schuh (1996) 
pointed out that measuring student satisfactions is a well- 
accepted practice in higher education. A university 
student survey is a useful data-gathering tool in that it 
provides institutions with valuable feedback regarding 
students' individual and collective impressions of the 
campus climate. Both individual and institutional factors 
are equally important. Identifying these characteristics 
helps administrators determine if institutions seek to 
improve the academic and student affairs in the higher 
education institution (Astin, 1993) . Also, examining both 
individual and institutional factors that impact student 
satisfaction is critical to improving student retention. 
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Recent higher education research suggested the 
application of hierarchical linear modeling (HLM) to the 
research on college student (Ethington, 1997) . HLM can 
allow the researchers to look at hierarchically structured 
data and interpret results without ignoring these 
structures (Roberts, 2000) . Without considering the nature 
of nested data structure from multiple institutions (where 
students are nested within institutions) , the conclusions 
from the conventional regression analysis could be 
inaccurate or even misleading. 

The present study presents an example using a 
university student satisfaction survey data to demonstrate 
how to address statistical problems associated with the 
hierarchical or nested nature of the data. The purpose of 
this paper is simply to illustrate the need for multilevel 
modeling (or HLM) , not to provide an extensive description 
of the usage of multilevel models. This study has two 
major limitations. First, only a few variables were 
included in the model. Second, very limited principles of 
the multilevel technique were engaged and described. For 
an extensive description of multilevel modeling procedures, 
see Snijders and Bosker (1999), Kreft and de Leeuw (1997), 
Goldstein (1995), and Bryk and Raudenbush (1992). Roberts 




7 



HLM & HE RESEARCH 7 



(in press) provides an excellent primer on multilevel 
modeling . 

Conceptual Framework and Variables 

This study is based on the belief that student 
satisfaction is understood best as a multilevel phenomenon 
of students and ma j or/department s . Level-1 unit of 
students will be nested within level-2 unit of 
maj or/department . Because of the recognition of this 
nested structure, HLM rather than multiple regression 
analysis is the appropriate technique to use (Bryk & 
Raudenbush, 1992; Ethington, 1997). 

Even within an institution, different majors/ 
departments may produce different influences on students. 
The environment within the maj or/department should have a 
more immediate influence on students than that of the 
institution as a whole. Holland (1966; 1985) suggested 
that individuals create their common environment as a 
function of their similar characteristics. This proximal 
environment of students can capture the immediate 
influences on students. 

Student satisfaction at the individual level is a 
function of the characteristics and experiences of 
individual students within their majors/departments. 
Student satisfaction at the maj or/departmental level is a 
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function of the characteristics of the ma j or/department and 
their impact on the individual experiences of the students 
within that major/department . 

The dependent variable in this study is the student 
satisfaction of major faculty within a maj or /department . 
Because this study is a multilevel study, independent 
variables at both the individual and maj or/departmental 
levels were included since satisfaction is believed to be a 
function of both sources of variance. The individual-level 
variable was used to estimate the effects of student 
background characteristics on satisfaction within 
individual majors/departments. In this study, student's 
classification (i.e., undergraduate and graduate) is the 
individual-level (level-1) variable. An institution-level 
variable was used to estimate the effects of 
maj or/departmental -level characteristics on differences in 
satisfaction between majors/departments. In this study, 
the number of faculty members in each maj or/department was 
used . 

Data Sources 

In the present study, self-report survey data was 
analyzed as a multilevel or hierarchically structured 
dataset. The study included student-level data, reflecting 
student background and satisfaction information, and 
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institutional-level data, showing student composition, or 
size. The University Student Survey, developed by 
University Planning of the University of North Texas, 
provided the student-level data. The Vice President Office 
of Academic Affairs provided the department - level data. 

Data Structure and Models 
The level -1 units are individual students and the 
level -2 units are their departments. Such a hierarchy is 
described in terms of clusters of level -1 units within each 
level -2 unit. Ignored clustering may cause standard errors 
of regression coefficients to be underestimated (Hox, 

1998). In the multilevel model, neither individual 
students nor individual departments are of primary 
interest. The main focus of this multilevel analysis is on 
estimating the pattern of variation in the underlying 
population (in this study, major or departments) (Rasbash 
et al . , 2001) . 

Multilevel modeling is an extension of OLS regression. 
This study uses two HLM models: (1) random intercept model 

and (2) random slope model (Rasbash et al . , 2001) . 

The random- intercepts model 

This analysis includes both student - level and 
institutional-level variables. The student - level model 
(level-1) use the student's characteristics to predict 
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outcomes on the dependent variable. The following equation 
is estimated for each institution 



y,j = Po x o + P\ x uj +u oj+% . 



which can also be written as 



and 



y,i = Pay * o + P\ x uj ' 



Poij Po "*■ U 0 j y e 0ii > 



ID 



( 2 ) 



13) 



where var(M 0/ ) = <j 2 u0 and var(e 0 , y ) = <j ] 0 . In this model, 
institutions vary only in their intercept. Thus, if x 0 is 
allowed to be random at both level -1 and level -2 in 
Equation 1, MLwiN added subscripts i and j to the 
coefficient J3 0 in Equations 2 and 3 . Each of the student- 

level predictors is centered about the institutional mean. 
The intercepts represent the institution mean dependent 
variable levels. The level -2 residual u 0 modifies the 
intercept term, but the slope coefficient /?, is fixed 
(Rasbash et al . , 2001, p. 33). 

The random intercepts and slopes model 

The random slopes model used in this study will be 

y>j = Po * 0 + Px X Uj + U \j + U 0j + e 0ij ' ( 4 ) 



or 



y 9 = Poy x o + P.i x uj > 
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fioij ~ Po +U 0j + e Oij » (^) 

and 

Pj = P+%> (7) 

where var(w oy ) = cr; i0 , var (u Uj ) = a 2 , , cow(u 0j ,u v ) = cr u01 , and var(e 0 , y ) = a 2 e0 . 
If x, is random at level-2, the subscript "j" should be 
added to the coefficient for x, . The level -2 residuals u 0j 
and u }J modifies both the intercept term /?„ and the slope 

coefficient /?, to vary (Rasbash et al . , 2001, p. 36) . 

The random intercepts and slopes model is probably the 
most realistic model to consider in this type of analysis. 
Considering that it would be erroneous to conclude that the 
slopes and intercepts for each maj or/department would be 
the same, we must model in the data the perceived 
variability about each of these coefficients (Roberts, in 
press) . 

Intraclass correlation (ICC) 

One of the important statistics in multilevel models 
is ICC. If ICC exists, the traditional OLS model must be 
abandoned because the assumption of independent 
observations has been violated (Kreft and de Leeuw, 1998) . 
ICC is the proportion of total variance that is between the 
groups of the regression equation. ICC determines the 
amount of total variance due to the differences between 
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majors/departments in this study. The homogeineity of the 
clusters can be determined by partitioning the variance in 
the outcome measure into its within-cluster and between- 
cluster components (Thomas & Heck, 2001) . The partitioning 
is accomplished using the equivalent of a one-way ANOVA 
with random effects where the sample cluster variable (in 
this study majors/departments) is treated as a random 
factor with the number of majors in the first-stage sample 
(Ethington, 1997, p.175; Thomas & Heck, 2001, p. 526). ICC 
{p) for a two-level model is the proportion of group level 

variance from the total variance, where cr 2 u0 represents the 

level-2 variance and cr 2 0 represents the level-1 variance 

&u0 ■*" &e0 

As a result, as similarities among students within 
departments become more pronounced in the sample, estimate 
of variance and standard errors derived from the data 
become more biased (Muthen & Satorra, 1995) . Ignoring 
clustering sampling leads to conservative estimates of 
standard errors and increase the possibility for committing 
Type I errors in hypothesis testing (Muthen & Satorra, 

1995; Thomas & heck, 2001) . 

The presence of a high ICC would mean that the 
students within clusters tend to be homogeneous. If 
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students within departments are more similar than students 
in other departments, then the value of ICC will be high. 

If the ICC is high, the model should allow the explanatory 
variable to be random at level-2. 

If the ICC is large, the impact of cluster sampling 
can be substantial. If the ICC coefficient is minimal, the 
observations are nearly independent and traditional 
multiple regression analysis will provide accurate 
estimates of the parameters and standard errors. 

Findings 

The data consist of 1,187 students (undergraduate: 79% 
and graduate: 21%) in 49 majors on both dependent and 
independent variables. The statistical package used was 
MLwiN Version 1.10 developed by the staff at the Multilevel 
Models Project, Institute of Education (Rasbash, Browne, 
Healy, Cameron, & Charlton, 2000) . In order to compare 
results from the OLS regression and multilevel modeling, 
the data will be first fit with SPSS and then with MLwiN. 
The dependent variable used was the composite score 
measuring students' satisfaction on faculty in students' 
major. The independent variable used was a background 
variable— classification (i.e., undergraduate and graduate 
students) for the level -1 predictor and the number of 
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faculty members of each maj or/department for the level -2 
predictor . 

Multilevel models incorporate both random and fixed 
effects when nesting is an obvious consequence of 
multistage sampling or when nesting is a source of random 
variability (Bryk & Raudenbush, 1992; Raudenbush, 1988) . 

Using Equation 1, I specified that the coefficient of 
intercept is random at both level-1 and level-2. The model- 
specified related student satisfaction to classification. 
The regression coefficients, /? 0 and /?, , define the average 
line across all students in all maj or/departments . The 
model is made multilevel by allowing each major/ 
department's summary line to depart from the average line 
by an amount u 0j . The i ' th student in the j ' th 
maj or/department departs from its summary line by an amount 
e oij . This model fits a set of parallel straight lines for 

the different majors/departments. The slopes of the lines 
are all the same. The fitted value of the common slope is 
1.291 with a standard error of 0.012 and it is highly 
significant. On the other hand, the intercepts of the 
lines vary. Their mean is 14.381 with a standard error of 
0.162. The intercepts for the different majors/departments 
are the level -2 residuals u 0J and these are distributed 
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around their mean with a variance 0.510 (standard error 
0.210) . The simple comparison with the standard error and 
also the use of the "interval and tests" procedures 
provides approximations that can act as rough guides 
(Rasbash et al . , 2000) . The actual data points do not fit 
exactly on the straight lines; they vary about them with 
the level-1 residuals e QiJ 10.679 (standard error 0.447). In 

addition, when I added the 2-level predictor, the fitted 
value of the common slope was -0.026 with a standard error 
of 0.017 and it was not significant. 

The homogeneity of clusters is measured by calculating 
an ICC coefficient. The degree of bias in estimating 
variance in the data collected through cluster samples is a 
function of the ICC present in the data: 

= 0511 = o 046 ( 4 6 o /o) 

10.678 + 0.511 

If students are taken at random from the population, their 
variance would be the sum of the level 2 and level 1 
variances, 0.510 + 10.679 = 11.189. The between 
ma j or/department variance makes up a population 0.046 of 
this total variance. The ICC in this dataset is not large. 
The presence of a small ICC means that there is a small 
dependence of context (in this case majors/departments) for 
individual student scores. 
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The ICC is examined in order to determine if further 
model modification needs to be made. A high correlation 
could indicate the need to free a variable at the highest 
level of the hierarchy (in this case level-2) . That is, 
there is a possibility that the majors/departments lines 
may have different slopes (In fact, there is a very low 
chance of random slopes in this study because the ICC is a 
minimal value) . 

Using Equation 4, I specified a coefficient of 
classification, which is random at level 2 . The 
coefficient of classification had a suffix j indicating 
that it varies from maj or/department to maj or/department . 
The mean is 1.290 (standard error 0.264), not far different 
from the model with a single slope. The individual 
maj or /department slopes vary about this mean with a 
variance estimated as 0.511 (standard error 0.211). The 
intercepts of the individual maj or/department lines also 
differ. Their mean is 14.381 (standard error 0.162). In 
addition there is zero covariance between intercepts and 
slopes. It suggests that maj or/department with higher 
intercepts has nothing to do with steeper slopes and this 
corresponds to zero correlation between the intercept and 
slope across majors/departments. As students' individual 
scores vary around their majors/departments lines by 




17 



HLM & HE RESEARCH 1 7 



quantities e 0ij and their variance is 10.678 (standard error 
0.446) . 



The quantity of -2*log-likelihood can be used to make 
an overall comparison of this more complicated model with 
the previous one. It has remained the same. The new model 
involves two extra parameters; the variance of the slope 
residuals u and their covariance with the intercept 

residuals u 0j and the change can be regarded as a % 2 value 

with 2df under the null hypothesis that the extra 
parameters have population values of zero. It confirms no 
better fit of the more elaborate model to the data. 

Table 1 presents the results from the simple bivariate 
regression for both OLS and multilevel cases. In an OLS 
equation, the unstandardized coefficient for the slope of 
Table 1 

Results comparing the ordinary least squares model and the 



multilevel model. 


Variable 


OLS 

Estimate 


Muti level 
Estimate 


Greater 

Value 


Intercept 


14.396 


14.381 


OLS 




(0.109) 


(0.162) 




Dummy variable 


1.418 


1.290 


OLS 


Undergraduate 


(0.239) 


(0.264) 




Vs . Graduate 








Number of Fac 


-0 . 0272 


-0 . 026 


Mult 




(0.012) 


(0 . 017) 




Note: Multiple 


R 2 for the OLS 


model is 0.029. 





Standard error in parenthesis 
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classification was 1.418. The results mean that students' 
satisfaction may vary across the classification of students 
(in this study, here undergraduate versus graduate 
students) . 

Contrasting with the OLS equation, a multilevel model 
was used to fit the data and to compare weights from the 
OLS regression and multilevel model. When solved for in 
the multilevel model equation, beta weight was 1.290. This 
is the interpretation of results from the OLS regression. 
The result from the multilevel model indicates that the 
these two procedures provided the similar coefficient of 
slope for the variable "classification." Therefore, both 
the OLS equation and the multilevel model may indicate that 
students' satisfaction scores is the function of the 
classification . 

Conclusion 

The purpose of this study is to illustrate the need 
for multilevel modeling in higher education research when 
the data are hierarchically structured or nested. 

Multilevel modeling is a relatively new technique in the 
higher education research. The results from this study 
have implications for the current issues on institutional 
quality and for the current movement of educational 
improvement in higher education institutions. 
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OLS techniques may provide incorrect estimates in some 
datasets. Multilevel modeling techniques are most useful 
when intraclass correlation is large. A large ICC does not 
mean that the parameter estimates obtained from multilevel 
modeling will be quite different from OLS estimates but 
that there is a greater dependence of context (in this 
study, majors/department) for individual student scores. 
However, multilevel techniques are often more difficult to 
interpret and thus less generalizable (Roberts, 2000) . 

This paper used very limited source of dataset. 

Beyond the statistical issues are data issues. Only a few 
variables from a limited dataset were included in the 
model. The multiple levels should be adequately reflected 
in the data gathered and the structure of data should be 
adequately addressed. 

The special issue of the Journal of Educational and 
Behavioral Statistics (JEBS) (1995) provided problems and 
prospects of hierarchical linear models. In this special 
issue. Draper (1995) and de Leeuw and Kreft (1995) warned 
the uncritical use of modeling. Morris (1995) pointed out 
that when the number of level -2 units is small, the maximum 
likelihood methods do not provide better variance 
estimates. Raudenbush (1995) noted that even the level -2 
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coefficients can be misleading when the number of subjects 
varies substantially between units. 

Goldstein (1995) summed up the future of multilevel 
modeling : 

There is a danger, and this paper reminds us of it, 
that multilevel modeling will become so fashionable 
that its use will be a requirement of journal editors, 
or even worse, that the mere fact of having fitted a 
multilevel model will become a certificate of 
statistical probity. That would be a great pity. 

These models are as good as the data they fit: they 
are powerful tools, not universal panaceas (p. 202) . 
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